What is claimed is: 

1 . A video image synthesis method comprising the steps 

of: 

sampling two contiguous frames from a video image; 
placing a reference patch comprising one or a plurality 
of rectangular areas on one of said two frames which is used 
as a reference frame, then placing on the other of said two frames 
a second patch which is the same as said reference patch, then 
moving and/or deforming said second patch in said other frame 
so that an image within said second patch coincides with an image 
within said reference patch, and estimating a correspondent 
relationship between a pixel within said second patch on said 
other frame and a pixel within said reference patch on said 
reference frame, based on said second patch after the movement 
and/or deformation and on said reference patch; 

acquiring a first interpolated frame whose resolution 
is higher than each of said frames, by performing interpolation 
either on the image within said second patch of said other frame 
or on the image within said second patch of said other frame 
and image within said reference patch of said reference frame, 
based on said correspondent relationship; 

acquiring a second interpolated frame whose resolution 
is higher than each of said frames, by performing interpolation 
on the image within said reference patch of said reference frame; 

acquiring a coordinate-transformed frame by 
transforming coordinates of the image within said second patch 
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of said other frame to a coordinate space of said reference frame, 
based on said correspondent relationship; 

computing a correlation value that represents a 
correlation between the image within the patch of said 
5 coordinate-transformed frame and the image within said reference 

patch of said reference frame; 

acquiring a weighting coefficient that makes a weight 
of said first interpolated frame greater as said correlation 
becomes greater, when synthesizing said first interpolated frame 
10 and second interpolated frame, based on said correlation value; 

and 

acquiring a synthesized frame by weighting and 
synthesizing said first and second interpolated frames, based 
on said weighting coefficient. 
15 2 . A video image synthesis method comprising the steps 



of: 



image; 



sampling three or more contiguous frames from a video 



placing a reference patch comprising one or a plurality 
20 of rectangular areas on one of said three or more frames which 

is used as a reference frame, then respectively placing on the 
others of said three or more frames patches which are the same 
as said reference patch, then moving and/or deforming saidpatches 
in said other frames so that an image within the patch of each 
25 of said other frames coincides with an image within said reference 

patch, and respectively estimating correspondent relationships 
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between pixels within the patches of said other frames and a 
pixel within said reference patch of said reference frame, based 
on the patches of said other frames after the movement and/or 
deformation and on said reference patch; 

acquiring a plurality of first interpolated frames 
whose resolution is higher than each of said frames, by performing 
interpolation either on the image within the patch of each of 
said other frames or on the image within the patch of each of 
said other frames and image within said reference patch of said 
reference frame, based on said correspondent relationships; 

acquiring one or a plurality of second interpolated 
frames whose resolution is higher than each of said frames and 
which are correlated with said plurality of first interpolated 
frames, by performing interpolation on the image within said 
reference patch of said reference frame; 

acquiring a plurality of coordinate-transformed 
frames by transforming coordinates of the images within the 
patches of saidother frames to a coordinate space of said reference 
frame, based on said correspondent relationships; 

computing correlation values that represent a 
correlation between the image within the patch of each of said 
coordinate-transformed frames and the image within said reference 
patch of said reference frame; 

acquiring weighting coefficients that make a weight 
of said first interpolated frame greater as said correlation 
becomes greater, when synthesizing said first interpolated frame 
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and second interpolated frame, based on said correlation values; 
and 

acquiring intermediate synthesized frames by 
weighting and synthesizing said first and second interpolated 
frames that correspond to each other on the basis of said weighting 
coefficients, and acquiring a synthesized frame by synthesizing 
said intermediate synthesized frames. 

3. The synthesis method as set forth in claim 1, wherein 
when said correlation value has been computed for each 
of the pixels and/or each local region that constitute each of 
said frames, 

saidcorrelation value is filtered to compute a filtered 
correlation value, and said weighting coefficient is acquired 
based on said filtered correlation value. 

4. The synthesis method as set forth in claim 2 , wherein 
when said correlation value has been computed for each 
of the pixels and/or each local region that constitute each of 
said frames, 

said correlation value is filtered to compute a filtered 
correlation value, and said weighting coefficient is acquired 
based on said filtered correlation value. 

5. The synthesis methodas set forth in claim 1 , wherein 
when said correlation value has been computed for each 
of the pixels and/or each local region that constitute each of 
25 said frames, 

said weighting coefficient is interpolated to acquire 
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20 
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weighting coefficients for all pixels that constitute said first 
and second interpolated frames. 

6 . The synthesis method as set forth in claim 2, wherein 
when said correlation value has been computed for each 

of the pixels and/or each local region that constitute each of 
said frames, 

said weighting coefficient is interpolated to acquire 
weighting coefficients for all pixels that constitute said first 
and second interpolated frames. 

7 . The synthesis methodas set forth in claim 1, wherein 
said weighting coefficient is acquiredby referring to a nonlinear 
graph in which said correlation value is represented in the 
horizontal axis and said weighting coefficient in the vertical 
axis . 

8. The synthesis methodas set forth in claim 2, wherein 
said weighting coefficient is acquiredby referring to a nonlinear 
graph in which said correlation value is represented in the 
horizontal axis and said weighting coefficient in the vertical 
axis . 

9. The synthesis methodas set forth in claim 1 , wherein 
the estimation of said correspondent relationship, acquisition 
of said first interpolated frame, acquisition of said second 
interpolated frame, acquisition of said coordinate-transformed 
frame, computation of said correlation value, acquisition of 
said weighting coefficient, and acquisition of said synthesized 
frame are performed by employing at least one component that 



constitutes said frame. 

10. The synthesis method as set forth in claim 2, 
wherein the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of 
said second interpolated frame, acquisition of said 
coordinate-transformed frame, computation of said correlation 
value, acquisitionof said weighting coefficient, and acquisition 
of said synthesized frame are performed by employing at least 
one component that constitutes said frame. 

11. A video image synthesizer comprising: 
sampling means for sampling two contiguous frames from 

a video images- 
correspondent relationship estimation means for 
placing a reference patch comprising one or a plurality of 
rectangular areas on one of said two frames which is used as 
a reference frame, then placing on the other of said two frames 
a second patch which is the same as said reference patch, then 
moving and/or deforming said second patch in said other frame 
so that an image within said second patch coincides with an image 
within said reference patch, and estimating a correspondent 
relationship between a pixel within said second patch on said 
other frame and a pixel within said reference patch on said 
reference frame, based on said second patch after the movement 
and/or deformation and on said reference patch; 

first interpolation means for acquiring a first 
interpolated frame whose resolution is higher than each of said 



frames, by performing interpolation either on the image within 
said second patch of said other frame or on the image within 
said second patch of said other frame and image within said 
reference patch of said reference frame, based on said 
5 correspondent relationship; 

second interpolation means for acquiring a second 
interpolated frame whose resolution is higher than each of said 
frames, by performing interpolation on the image within said 
reference patch of said reference frame; 
10 coordinate transformation means for acquiring a 

coordinate-transformed frame by transforming coordinates of the 
image within said second patch of said other frame to a coordinate 
space of said reference frame, based on said correspondent 
relationship; 

15 correlation-value computation means for computing a 

correlation value that represents a correlation between the image 
within the patch of said coordinate-transformed frame and the 
image within said reference patch of said reference frame; 

weighting-coefficient acquisit ionmeans for acquiring 

20 a weighting coefficient that makes a weight of said first 

interpolated frame greater as said correlation becomes greater, 
when synthesizing said first interpolated frame and second 
interpolated frame, based on said correlation value; and 

synthesis means for acquiring a synthesized frame by 

25 weighting and synthesizing said first and second interpolated 

frames, based on said weighting coefficient. 
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12. A video image synthesizer comprising: 
sampling means for sampling three or more contiguous 
frames from a video image; 

correspondent relationship estimation means for 
5 placing a reference patch comprising one or a plurality of 

rectangular areas on one of said three or more frames which is 
used as a reference frame, then respectively placing on the others 
of said three or more frames patches which are the same as said 
reference patch, then moving and/or deforming said patches in 
10 said other frames so that an image within the patch of each of 

said other frames coincides with an image within said reference 
patch, and respectively estimating correspondent relationships 
between pixels within the patches of said other frames and a 
pixel within said reference patch of said reference frame, based 
15 on the patches of said other frames after the movement and/or 

deformation and on said reference patch; 

first interpolation means for acquiring a plurality 
of first interpolated frames whose resolution is higher than 
each of said frames, by performing interpolation either on the 
20 image within the patch of each of said other frames or on the 

image within the patch of each of said other frames and image 
within said reference patch of said reference frame, based on 
said correspondent relationships; 

second interpolation means for acquiring one or a 
25 plurality of second interpolated frames whose resolution is 

higher than each of said frames and which are correlated with 
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said plurality of first interpolated frames, by performing 
interpolation on the image within said reference patch of said 
reference frame; 

coordinate transformation means for acquiring a 
5 plurality of coordinate-transformed frames by transforming 

coordinates of the images within the patches of said other frames 
to a coordinate space of said reference frame, based on said 
correspondent relationships ; 

correlation-value computation means for computing 
10 correlation values that represent a correlation between the image 

within the patch of each of said coordinate-transformed frames 
and the image within said reference patch of said reference frame; 

weighting-coefficient acquisition means for acquiring 
weighting coefficients that make a weight of said first 
15 interpolated frame greater as said correlation becomes greater, 

when synthesizing said first interpolated frame and second 
interpolated frame, based on said correlation values; and 

synthesis means for acquiring intermediate 
synthesized frames by weighting and synthesizing said first and 
20 second interpolated frames that correspond to each other on the 

basis of said weighting coefficients , and acquiring a synthesized 
frame by synthesizing said intermediate synthesized frames. 

13. The video image synthesizer as set forth in claim 
11, wherein 

25 when said correlation value has been computed for each 

of pixels and/or each local region that constitute each of said 
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frames, 

said synthesizer further comprises means for filtering 
said correlation value to compute a filtered correlation value, 
and said weighting-coefficient acquisition means acquires said 
weighting coefficient, based on said filtered correlation value . 

14 . The video image synthesizer as set forth in claim 
12, wherein 

when said correlation value has been computed for each 
of pixels and/or each local region that constitute each of said 
frames, 

said synthesizer further comprises means for filtering 
said correlation value to compute a filtered correlation value, 
and said weighting-coefficient acquisition means acquires said 
weighting coefficient, based on said filtered correlation value . 

15. The video image synthesizer as set forth in claim 

11, wherein 

when said correlation value has been computed for each 
of the pixels and/or each local region that constitute each of 
said frames, 

said weighting-coefficient acquisition means performs 
interpolation on said weighting coefficient, thereby acquiring 
weighting coefficients for all pixels that constitute said first 
and second interpolated frames . 

16. The video image synthesizer as set forth in claim 

12, wherein 

when said correlation value has been computed for each 
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of the pixels and/or each local region that constitute each of 
said frames, 

said weighting-coefficient acquisition means performs 
interpolation on said weighting coef f icient, thereby acquiring 
5 weighting coefficients for all pixels that constitute said first 

and second interpolated frames. 

17 . The video image synthesizer as set forth in claim 

11, wherein said weighting-coefficient acquisition means 
acquires said weighting coefficient by referring to a nonlinear 

10 graph in which said correlation value is represented in the 

horizontal axis and said weighting coefficient in the vertical 
axis . 

18. The video image synthesizer as set forth in claim 

12, wherein said weighting-coefficient acquisition means 

15 acquires said weighting coefficient by referring to a nonlinear 

graph in which said correlation value is represented in the 
horizontal axis and said weighting coefficient in the vertical 
axis . 

19. The video image synthesizer as set forth in claim 
20 11, wherein said correspondent relationship estimation means, 

said first interpolation means, said second interpolation means, 
said coordinate transformation means, said correlation-value 
computation means, said weighting-coefficient acquisition means, 
and said synthesis means perform the estimation of said 
25 correspondent relationship, acquisition of said first 

interpolated frame, acquisition of said second interpolated frame, 
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acquisition of said coordinate-transformed frame, computation 
of said correlation value, acquisition of said weighting 
coefficient, and acquisition of said synthesized frame, by 
employing at least one component that constitutes said frame. 
5 20. The video image synthesizer as set forth in claim 

12, wherein said correspondent relationship estimation means, 
said first interpolation means, said second interpolation means, 
said coordinate transformation means, said correlation-value 
computation means , said weighting-coefficient acquisition means , 

10 and said synthesis means perform the estimation of said 

correspondent relationship, acquisition of said first 
interpolated frame, acquisition of said second interpolated frame, 
acquisition of said coordinate-transformed frame, computation 
of said correlation value, acquisition of said weighting 

15 coefficient, and acquisition of said synthesized frame, by 

employing at least one component that constitutes said frame. 

21 . A program for causing a computer to execute a video 
image synthesis method comprising: 

a procedure of sampling two contiguous frames from a 

2 0 video image; 

a procedure of placing a reference patch comprising 
one or a plurality of rectangular areas on one of said two frames 
which is used as a reference frame, then placing on the other 
of said two frames a secondpatch which is the same as said reference 

25 patch, then moving and/or deforming said second patch in said 

other frame so that an image within said second patch coincides 
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with an image within said reference patch, and estimating a 
correspondent relationship between a pixel within said second 
patch on said other frame and a pixel within said reference patch 
on said reference frame, based on said second patch after the 
movement and/or deformation and on said reference patch; 

a procedure of acquiring a first interpolated frame 
whose resolution is higher than each of said frames , by performing 
interpolation either on the image within said second patch of 
said other frame or on the image within said second patch of 
said other frame and image within said reference patch of said 
reference frame, based on said correspondent relationship; 

a procedure of acquiring a second interpolated frame 
whose resolution is higher than each of said frames, by performing 
interpolation on the image within said reference patch of said 
reference frame; 

a procedure of acquiring a coordinate-transformed 
frame by transforming coordinates of the image within said second 
patch of said other frame to a coordinate space of said reference 
frame, based on said correspondent relationship; 

a procedure of computing a correlation value that 
represents a correlation between the image within the patch of 
said coordinate-transformed frame and the image within said 
reference patch of said reference frame; 

a procedure of acquiring a weighting coefficient that 
makes a weight of said first interpolated frame greater as said 
correlation becomes greater, when synthesizing said first 
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interpolated frame and second interpolated frame, based on said 
correlation value; and 

a procedure of acquiring a synthesized frame by 
weighting and synthesizing said first and second interpolated 
5 frames, based on said weighting coefficient. 

22 . A program for causing a computer to execute a video 
image synthesis method comprising: 

a procedure of sampling three or more contiguous frames 
from a video image; 

10 a procedure of placing a reference patch comprising 

one or a plurality of rectangular areas on one of said three 
or more frames which is used as a reference frame, then respectively 
placing on the others of said three or more frames patches which 
are the same as said reference patch, then moving and/or deforming 

15 said patches in said other frames so that an image within the 

patch of each of said other frames coincides with an image within 
said reference patch, and respectively estimating correspondent 
relationships between pixels within the patches of said other 
frames and a pixel within said reference patch of said reference 

2 0 frame, based on the patches of said other frames after the movement 

and/or deformation and on said reference patch; 

a procedure of acquiring a plurality of first 
interpolated frames whose resolution is higher than each of said 
frames, by performing interpolation either on the image within 

25 the patch of each of said other frames or on the image within 

the patch of each of said other frames and image within said 
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reference patch of said reference frame, based on said 
correspondent relationships ; 

a procedure of acquiring one or a plurality of second 
interpolated frames whose resolution is higher than each of said 
frames and which are correlated with said plurality of first 
interpolated frames, by performing interpolation on the image 
within said reference patch of said reference frame; 

a procedure of acquiring a plurality of 
coordinate-transformed frames by transforming coordinates of 
the images within the patches of said other frames to a coordinate 
space of said reference frame, based on said correspondent 
relationships ; 

a procedure of computing correlation values that 
represent a correlation between the image within the patch of 
each of said coordinate-transformed frames and the image within 
said reference patch of said reference frame; 

a procedure of acquiring weighting coefficients that 
make a weight of said first interpolated frame greater as said 
correlation becomes greater, when synthesizing said first 
interpolated frame and second interpolated frame, based on said 
correlation values; and 

a procedure of acquiring intermediate synthesized 
frames by weighting and synthesizing said first and second 
interpolated frames that correspond to each other on the basis 
of said weighting coef f icients , and acquiring a synthesized frame 
by synthesizing said intermediate synthesized frames. 
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23. The program as set forth in claim 21, wherein 
when said correlation value has been computed for each 

pixel and/or each local region that constitutes each of said 

frames, 

5 said method further comprises a procedure of filtering 

said correlation value to compute a filtered correlation value, 
and said weighting-coefficient acquisition procedure is a 
procedure of acquiring said weighting coefficient, based on said 
filtered correlation value. 
10 24. The program as set forth in claim 22, wherein 

when said correlation value has been computed for each 
pixel and/or each local region that constitute each of said frames, 
said method further comprises a procedure of filtering 
said correlation value to compute a filtered correlation value, 
15 and said weighting-coefficient acquisition procedure is a 

procedure of acquiring said weighting coefficients, based on 
said filtered correlation value. 

25. The program as set forth in claim 21, wherein 
when said correlation value has been computed for each 

20 of the pixels and/or each local region that constitute each of 

said frames, 

said weighting-coefficient acquisition procedure is 
a procedure of performing interpolation on said weighting 
coefficient and acquiring weighting coefficients for all pixels 
25 that constitute said first and second interpolated frames. 

26. The program as set forth in claim 22, wherein 
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when said correlation value has been computed for each 
of the pixels and/or each local region that constitute each of 
said frames, 

said weighting-coefficient acquisition procedure is 
a procedure of performing interpolation on said weighting 
coefficients and acquiring weighting coefficients for all pixels 
that constitute said first and second interpolated frames. 

27 . The program as set forth in claim 21, wherein said 
weighting-coefficient acquisition procedure is a procedure of 
acquiring said weighting coefficient by referring to a nonlinear 
graph in which said correlation value is represented in the 
horizontal axis and said weighting coefficient in the vertical 
axis. 

28 . The program as set forth in claim 22 , wherein said 
weighting-coefficient acquisition procedure is a procedure of 
acquiring said weighting coefficients by referring to a nonlinear 
graph in which said correlation value is represented in the 
horizontal axis and said weighting coefficient in the vertical 
axis . 

29 . The program as set forth in claim 21, wherein said 
correspondent relationship estimation procedure, said first 
interpolated frame acquisition procedure, said second 
interpolated frame acquisition procedure, said 
coordinate-transformed frame acquisition procedure, said 
correlation value computation procedure, said 
weighting-coefficient acquisition procedure, and said 



synthesized-f rame acquisition procedure are procedures of 
performing the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of 
said second interpolated frame, acquisition of said 
5 coordinate-transformed frame, computation of said correlation 

value, acquisition of saidweighting coefficient, and acquisition 
of said synthesized frame by employing at least one component 
that constitutes said frame, respectively. 

30 . The program as set forth in claim 22, wherein said 

10 correspondent relationship estimation procedure, said first 

interpolated frame acquisition procedure, said second 
interpolated frame acquisition procedure, said 
coordinate-transformed frame acquisition procedure, said 
correlation value computation procedure, said 

15 weighting-coefficient acquisition procedure, and said 

synthesized-f rame acquisition procedure are procedures of 
performing the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of 
said second interpolated frame, acquisition of said 

20 coordinate-transformed frame, computation of said correlation 

value, acquisition of saidweighting coefficient, and acquisition 
of said synthesized frame by employing at least one component 
that constitutes said frame, respectively. 

31. A video image synthesis method comprising the 

25 steps of: 

sampling two contiguous frames from a video image; 
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placing a reference patch comprising one or a plurality 
of rectangular areas on one of said two frames which is used 
as a reference frame, then placing on the other of said two frames 
a second patch which is the same as said reference patch, then 
5 moving and/or deforming said second patch in said other frame 

so that an image within said second patch coincides with an image 
within said reference patch, and estimating a correspondent 
relationship between a pixel within said second patch on said 
other frame and a pixel within said reference patch on said 
10 reference frame, based on said second patch after the movement 

and/or deformation and on said reference patch; 

acquiring a first interpolated frame whose resolution 
is higher than each of said frames, by performing interpolation 
either on the image within said second patch of said other frame 
15 or on the image within said second patch of said other frame 

and image within said reference patch of said reference frame, 
based on said correspondent relationship; 

acquiring a second interpolated frame whose resolution 
is higher than each of said frames, by performing interpolation 
20 on the image within said reference patch of said reference frame; 

acquiring edge information that represents an edge 
intensity of the image within said reference patch of said 
reference frame and/or image within the patch of said other frame; 

acquiring a weighting coefficient that makes a weight 
25 of said first interpolated frame greater as said edge information 

becomes greater, when synthesizing said first interpolated frame 
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and second interpolated frame, based on said edge information; 
and 

acquiring a synthesized frame by weighting and 
synthesizing said first and second interpolated frames, based 
5 on said weighting coefficient. 

32. A video image synthesis method comprising the 

steps of: 

sampling three or more contiguous frames from a video 

image; 

10 placing a reference patch comprising one or a plurality 

of rectangular areas on one of said three or more frames which 
is used as a reference frame, then respectively placing on the 
others of said three or more frames patches which are the same 
as said reference patch, then moving and/ or deforming said patches 

15 in said other frames so that an image within the patch of each 

of said other frames coincides with an image within said reference 
patch, and respectively estimating correspondent relationships 
between pixels within the patches of said other frames and a 
pixel within said reference patch of said reference frame, based 

20 on the patches of said other frames after the movement and/or 

deformation and on said reference patch; 

acquiring a plurality of first interpolated frames 
whose resolution is higher than each of said frames, by performing 
interpolation either on the image within the patch of each of 

25 said other frames or on the image within the patch of each of 

said other frames and image within said reference patch of said 
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reference frame, based on said correspondent relationships; 

acquiring one or a plurality of second interpolated 
frames whose resolution is higher than each of said frames and 
which are correlated with said plurality of first interpolated 
5 frames, by performing interpolation on the image within said 

reference patch of said reference frame; 

acquiring edge information that represents an edge 
intensity of the image within said reference patch of said 
reference frame and/or image within the patch of each of said 
10 other frames; 

acquiring weighting coefficients that make a weight 
of said first interpolated frame greater as said edge information 
becomes greater, when synthesizing said first interpolated frame 
and second interpolated frame, based on said edge information; 
15 and 

acquiring intermediate synthesized frames by 
weighting and synthesizing said first and second interpolated 
frames that correspond to each other on the basis of said weighting 
coefficients, and acquiring a synthesized frame by synthesizing 
20 said intermediate synthesized frames. 

33. The synthesis method as set forth in claim 31," 

wherein 

when said edge information has been computed for each 
of the pixels that constitute each of said frames, 
25 said weighting coefficient is interpolated to acquire 

weighting coefficients for all pixels that constitute said first 
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and second interpolated frames. 

34. The synthesis method as set forth in claim 32, 

wherein 

when said edge information has been computed for each 
of the pixels that constitute each of said frames, 

said weighting coefficient is interpolated to acquire 
weighting coefficients for all pixels that constitute said first 
and second interpolated frames. 

35. The synthesis method as set forth in claim 31^ 
wherein the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of 
said second interpolated frame, acquisition of said edge 
information, acquisition of said weighting coefficient, and 
acquisition of said synthesized frame are performed by employing 
at least one component that constitutes said frame. 

36. The synthesis method as set forth in claim 32.; 
wherein the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of 
said second interpolated frame, acquisition of said edge 
information, acquisition of said weighting coefficient, and 
acquisition of said synthesized frame are performed by employing 
at least one component that constitutes said frame. 

37. A video image synthesizer comprising: 
sampling means for sampling two contiguous frames from 

a video image; 

correspondent relationship estimation means for 



placing a reference patch comprising one or a plurality of 
rectangular areas on one of said two frames which is used as 
a reference frame, then placing on the other of said two frames 
a second patch which is the same as said reference patch, then 
5 moving and/or deforming said second patch in said other frame 

so that an image within said second patch coincides with an image 
within said reference patch, and estimating a correspondent 
relationship between a pixel within said second patch on said 
other frame and a pixel within said reference patch on said 

10 reference frame, based on said second patch after the movement 

and/or deformation and on said reference patch; 

first interpolation means for acquiring a first 
interpolated frame whose resolution is higher than each of said 
frames, by performing interpolation either on the image within 

15 said second patch of said other frame or on the image within 

said second patch of said other frame and image within said 
reference patch of said reference frame, based on said 
correspondent relationship; 

second interpolation means for acquiring a second 

20 interpolated frame whose resolution is higher than each of said 

frames, by performing interpolation on the image within said 
reference patch of said reference frame; 

edge information acquisition means for acquiring edge 
information that represents an edge intensity of the image within 

25 said reference patch of said reference frame and/or image within 

the patch of said other frame; 
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weighting-coefficient acquisition means for acquiring 
a weighting coefficient that makes a weight of said first 
interpolated frame greater as said edge information becomes 
greater, when synthesizing said first interpolated frame and 
second interpolated frame, based on said edge information; and 

synthesis means for acquiring a synthesized frame by 
weighting and synthesizing said first and second interpolated 
frames, based on said weighting coefficient. 

38. A video image synthesizer comprising: 

sampling means for sampling three or more contiguous 
frames from a video image; 

correspondent relationship estimation means for 
placing a reference patch comprising one or a plurality of 
rectangular areas on one of said three or more frames which is 
used as a reference frame, then respectively placing on the others 
of said three or more frames patches which are the same as said 
reference patch, then moving and/or deforming said patches in 
said other frames so that an image within the patch of each of 
said other frames coincides with an image within said reference 
patch, and respectively estimating correspondent relationships 
between pixels within the patches of said other frames and a 
pixel within said reference patch of said reference frame, based 
on the patches of said other frames after the movement and/or 
deformation and on said reference patch; 

first interpolation means for acquiring a plurality 
of first interpolated frames whose resolution is higher than 
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each of said frames, by performing interpolation either on the 
image within the patch of each of said other frames or on the 
image within the patch of each of said other frames and image 
within said reference patch of said reference frame, based on 
5 said correspondent relationships; 

second interpolation means for acquiring one or a 
plurality of second interpolated frames whose resolution is 
higher than each of said frames and which are correlated with 
said plurality of first interpolated frames, by performing 
10 interpolation on the image within said reference patch of said 

reference frame; 

edge information acquisition means for acquiring edge 
information that represents an edge intensity of the image within 
said reference patch of said reference frame and/or image within 
15 the patch of each of said other frames; 

weighting-coefficient acquisition means for acquiring 
weighting coefficients that make a weight of said first 
interpolated frame greater as said edge information becomes 
greater, when synthesizing said first interpolated frame and 
20 second interpolated frame, based on said edge information; and 

synthesis means for acquiring intermediate 
synthesized frames by weighting and synthesizing said first and 
second interpolated frames that correspond to each other on the 
basis of said weighting coefficients, and acquiring a synthesized 
25 frame by synthesizing said intermediate synthesized frames. 

39- The video image synthesizer as set forth in claim 
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37, wherein 

when said edge information has been computed for each 
of the pixels that constitute each of said frames, 

said weighting-coefficient acquisition means performs 
5 interpolation on said weighting coefficient, thereby acquiring 

weighting coefficients for all pixels that constitute said first 
and second interpolated frames. 

40. The video image synthesizer as set forth in claim 

38, wherein 

10 when said edge information has been computed for each 

of the pixels that constitute each of said frames, 

said weighting-coefficient acquisition means performs 
interpolation on said weighting coefficients, thereby acquiring 
weighting coefficients for all pixels that constitute said first 

15 and second interpolated frames. 

41 . The video image synthesizer as set forth in claim 
37, wherein said correspondent relationship estimation means, 
said first interpolation means, said second interpolation means, 
said edge information acquisition means, said 

20 weighting-coefficient acquisition means, and said synthesis 

means perform the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of 
said second interpolated frame, acquisition of said edge 
information, acquisition of said weighting coefficient, and 

25 acquisition of said synthesized frame, by employing at least 

one component that constitutes said frame. 
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42. The video image synthesizer as set forth in claim 
38, wherein said correspondent relationship estimation means, 
said first interpolation means, said second interpolation means, 
said edge information acquisition means, said 
5 weighting-coefficient acquisition means, and said synthesis 

means perform the estimation of said correspondent relationship, 
acquisition of said first interpolated frame, acquisition of 
said second interpolated frame, acquisition of said edge 
information, acquisition of said weighting coefficient, and 

10 acquisition of said synthesized frame, by employing at least 

one component that constitutes said frame. 

43 . A program for causing a computer to execute a video 
image synthesis method comprising: 

a procedure of sampling two contiguous frames from a 

15 video image; 

a procedure of placing a reference patch comprising 
one or a plurality of rectangular areas on one of said two frames 
which is used as a reference frame, then placing on the other 
of said two frames a secondpatch which is the same as said reference 

20 patch, then moving and/or deforming said second patch in said 

other frame so that an image within said second patch coincides 
with an image within said reference patch, and estimating a 
correspondent relationship between a pixel within said second 
patch on said other frame and a pixel within said reference patch 

25 on said reference frame, based on said second patch after the 

movement and/or deformation and on said reference patch; 
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a procedure of acquiring a first interpolated frame 
whose resolution is higher than each of said frames, by performing 
interpolation either on the image within said second patch of 
said other frame or on the image within said second patch of 
5 said other frame and image within said reference patch of said 

reference frame, based on said correspondent relationship; 

a procedure of acquiring a second interpolated frame 
whose resolution is higher than each of said frames, by performing 
interpolation on the image within said reference patch of said 
10 reference frame; 

a procedure of acquiring edge information that 
represents an edge intensity of the image within said reference 
patch of said reference frame and/or image within the patch of 
said other frame; 

15 a procedure of acquiring a weighting coefficient that 

makes a weight of said first interpolated frame greater as said 
edge information becomes greater, when synthesizing said first 
interpolated frame and second interpolated frame, based on said 
edge information; and 

20 a procedure of acquiring a synthesized frame by 

weighting and synthesizing said first and second interpolated 
frames, based on said weighting coefficient. 

44 . Aprogram for causing a computer to execute a video 
image synthesis method comprising: 

25 a procedure of sampling three or more contiguous frames 

from a video image; 
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a procedure of placing a reference patch comprising 
one or a plurality of rectangular areas on one of said three 
or more frames which is usedas a reference frame, then respectively 
placing on the others of said three or more frames patches which 
are the same as said reference patch, then moving and/or deforming 
said patches in said other frames so that an image within the 
patch of each of said other frames coincides with an image within 
said reference patch, and respectively estimating correspondent 
relationships between pixels within the patches of said other 
frames and a pixel within said reference patch of said reference 
frame, based on the patches of said other frames after the movement 
and/or deformation and on said reference patch; 

a procedure of acquiring a plurality of first 
interpolated frames whose resolution is higher than each of said 
frames, by performing interpolation either on the image within 
the patch of each of said other frames or on the image within 
the patch of each of said other frames and image within said 
reference patch of said reference frame, based on said 
correspondent relationships ; 

a procedure of acquiring one or a plurality of second 
interpolated frames whose resolution is higher than each of said 
frames and which are correlated with said plurality of first 
interpolated frames, by performing interpolation on the image 
within said reference patch of said reference frame; 

a procedure of acquiring edge information that 
represents an edge intensity of the image within said reference 



168 



patch of said reference frame and/or image within the patch of 
each of said other frames; 

a procedure of acquiring weighting coefficients that 
make a weight of said first interpolated frame greater as said 
5 edge information becomes greater, when synthesizing said first 

interpolated frame and second interpolated frame, based on said 
edge information; and 

a procedure of acquiring intermediate synthesized 
frames by weighting and synthesizing said first and second 
10 interpolated frames that correspond to each other on the basis 

of said weighting coefficients, and acquiring a synthesized frame 
by synthesizing said intermediate synthesized frames. 

45. The program as set forth in claim 43, wherein 

when said edge information has been computed for each 
15 of the pixels that constitute each of said frames, 

said weighting-coefficient acquisition procedure is 
a procedure of performing interpolation on said weighting 
coefficient and acquiring weighting coefficients for all pixels 
that constitute said first and second interpolated frames. 
20 46. The program as set forth in claim 44, wherein 

when said edge information has been computed for each 
of the pixels that constitute each of said frames, 

said weighting-coefficient acquisition procedure is 
a procedure of performing interpolation on said weighting 
25 coefficients and acquiring weighting coefficients for all pixels 

that constitute said first and second interpolated frames. 
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47 . The program as set forth in claim 43, wherein said 
correspondent relationship estimation procedure, said first 
interpolated frame acquisition procedure, said second 
interpolated frame acquisition procedure, said edge information 

5 acquisition procedure, said weighting-coefficient acquisition 

procedure, and said synthesized-f rame acquisition procedure are 
procedures of performing the estimation of said correspondent 
relationship, acquisition of said first interpolated frame, 
acquisition of said second interpolated frame, acquisition of 
10 saidedge information, acquisition of said weighting coefficient , 

and acquisition of said synthesized frame by employing at least 
one component that constitutes said frame, respectively. 

48 . The program as set forth in claim 44, wherein said 
correspondent relationship estimation procedure, said first 

15 interpolated frame acquisition procedure, said second 

interpolated frame acquisition procedure, saidedge information 
acquisition procedure, said weighting-coefficient acquisition 
procedure, and said synthesized-f rame acquisition procedure are 
procedures of performing the estimation of said correspondent 

20 relationship, acquisition of said first interpolated frame, 

acquisition of said second interpolated frame, acquisition of 
saidedge information, acquisition of said weighting coefficient, 
and acquisition of said synthesized frame by employing at least 
one component that constitutes said frame, respectively. 

25 49. A video image synthesis method comprising the 

steps of: 
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sampling a predetermined number of contiguous frames, 
which include a reference frame and are two or more frames, from 
said video image; 

placing a reference patch comprising one or a plurality 
5 of rectangular areas on said reference frame; 

respectively placing patches which are the same as said 
reference patch, on the others of said predetermined number of 
frames; 

moving and/or deforming said patches in said other 
10 frames so that an image within the patch of each of said other 

frames approximately coincides with an image within said 

reference patch; 

respectively acquiring correspondent relationships 

between pixels within the patches of said other frames and a 
15 pixel within said reference patch of said reference frame, based 

on the patches of said other frames after the movement and/or 

deformation and on said reference patch; and 

acquiring a synthesized frame from said predetermined 

number of frames, based on said correspondent relationships; 
20 wherein said predetermined number of frames are 

determined based on image characteristics of said video image 

or synthesized frame, and said predetermined number of frames 

are sampled. 

50. The synthesis method as set forth in claim 49, 

25 wherein 

said correspondent relationships are acquired in order 
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of other frames closer to said reference frame, and a correlation 
is acquired between each of said other frames, in which said 
correspondent relationship is acquired, and saidref erence frame; 

and when said correlation is lower than a predetermined 
5 threshold value, acquisition of said correspondent relationships 

is stopped, and said synthesized frame is obtained based on said 
correspondent relationship by employing said other frames, in 
which said correspondent relationship has been acquired, and 
said reference frame. 

10 51. A video image synthesizer comprising: 

sampling means for sampling a predetermined number of 
contiguous frames, which include a reference frame and are two 
or more frames, from a video image; 

correspondent relationship acquisition means for 

15 placing a reference patch comprising one or a plurality of 

rectangular areas on said reference frame, then respectively 
placing on the others of said predetermined number of frames 
patches which are the same as said reference patch, then moving 
and/or deforming said patches in said other frames so that an 

20 image within the patch of each of said other frames approximately 

coincides with an image within said reference patch, and 
respectively acquiring correspondent relationships between 
pixels within the patches of said other frames and a pixel within 
said reference patch of said reference frame, based on the patches 

25 of said other frames after the movement and/or deformation and 

on said reference patch; and 



172 



frame synthesis means for acquiring a synthesized frame 
from said predetermined number of frames, based on said 
correspondent relationships acquired by said correspondent 
relationship acquisition means; 
5 wherein said sampling means is equipped with 

frame-number determination means for determining said 
predetermined number of frames on the basis of image 
characteristics of said video image or synthesized frame, and 
samples said predetermined number of frames determined by said 
10 frame-number determination means. 

52. The video image synthesizer as set forth in claim 
51, wherein 

said correspondent relationship acquisition means 
acquires said correspondent relationships in order of other 
15 frames closer to said reference frame; 

there is provided stoppage means for acquiring a 
correlation between each of said other frames, in which said 
correspondent relationship is acquired by said correspondent 
relationship acquisition means, and said reference frame, and 
20 stopping a process which is being performed in said correspondent 

relationship acquisition means when said correlation is lower 
than a predetermined threshold value; and 

said frame synthesis means acquires said synthesized 
frame by employing said other frames, in which said correspondent 
25 relationship has been acquired, and said reference frame, based 

on said correspondent relationship acquired by said correspondent 
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relationship acquisition means. 

53. A program for causing a computer to execute: 

a determination process of determining the number of 
frames, based on image characteristics of a video image or image 
5 characteristics of a synthesized frame which is obtained from 

the determined number of frames of said video image; 

a sampling process of sampling the determined number 
of frames which are contiguous and include a reference frame, 
from said video image; 

10 a correspondent relationship acquisition process of 

placing a reference patch comprising one or a plurality of 
rectangular areas on said reference frame, then respectively 
placing on the others of said predetermined number of frames 
patches which are the same as said reference patch, then moving 

15 and/or deforming said patches in said other frames so that an 

image within the patch of each of said other frames approximately 
coincides with an image within said reference patch, and 
respectively acquiring correspondent relationships between 
pixels within the patches of said other frames and a pixel within 

20 said reference patch of said reference frame, based on the patches 

of said other frames after the movement and/or deformation and 
on said reference patch; and 

a frame synthesis process of acquiring said synthesized 
frame from said determined number of frames, based on said 

25 correspondent relationships. 

54. The program as set forth in claim 53, wherein 
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said correspondent relationship acquisition process 
acquires said correspondent relationships in order of other 
frames closer to said reference frame; and 

said computer is further caused to execute a process 
of acquiring a correlation between each of said other frames, 
in which said correspondent relationship is acquired, and said 
reference frame, and stopping said correspondent relationship 
acquisition process when said correlation is lower than a 
predetermined threshold value. 

55. A video image synthesis method comprising the 

steps of: 

obtaining a contiguous frame group by detecting a 
plurality of frames that represent contiguous scenes in a video 
image; 

placing a reference patch comprising one or a plurality 
of rectangular areas on one of said plurality of frames included 
in said contiguous frame group which is used as a reference frame; 

respectively placing patches which are the same as said 
reference patch, on the others of said plurality of frames; 

moving and/or deforming said patches in said other 
frames so that an image within the patch of each of said other 
frames approximately coincides with an image within said 
reference patch; 

respectively acquiring correspondent relationships 
between pixels within the patches of said other frames and a 
pixel within said reference patch of said reference frame, based 



on the patches of said other frames after the movement and/or 
deformation and on said reference patch; and 

acquiring a synthesized frame from said plurality of 
frames, based on said correspondent relationships. 

56. The synthesis method as set forth in claim 55, 

wherein 

a correlation between adjacent frames, which is started 
from said reference frame, is acquired; and 

said contiguous frame group that is detected comprises 
frames ranging from said reference frame to a frame, which is 
closer to said reference frame, between a pair of said adjacent 
frames in which said correlation is lower than a predetermined 
first threshold value. 

57. The synthesis method as set forth in claim 56, 

wherein 

a histogram is computed for at least one of the Y, Cb, 
and Cr components of each of said adjacent frames (where the 
Y component is a luminance component and the Cb and Cr components 
are color difference components) ; 

a Euclidean distance for each component between said 
adjacent frames is computed by employing said histogram; 

the sum of the Euclidean distances for said three 
components is computed; and 

when said sum is a predetermined second threshold value 
or greater, the correlation between said adjacent frames is lower 
than said predetermined first threshold value. 



58. The synthesis method as set forth in claim 57, 
wherein said histogram is computed by dividing each of components, 
which are used, among said three components by a value greater 
than 1 . 

5 59. The synthesis method as set forth in claim 56, 

wherein 

a difference between pixel values of corresponding 
pixels of said adjacent frames is computed for all corresponding 
pixels; 

10 the sum of absolute values of the differences for all 

corresponding pixels is computed; and 

when said sum is a third threshold value or greater, 

said correlation between adjacent frames is lower than said 

predetermined first threshold value. 
15 60. The synthesis method as set forth in claim 56, 

wherein said correlation is computed by employing a reduced image 

or thinned image of each frame. 

61. The synthesis method as set forth in claim 55, 
wherein the detection of frames that constitute said contiguous 

20 frame group is stopped when the number of detected frames reaches 

a predetermined upper limit value. 

62. A video image synthesizer comprising: 
contiguous frame group detection means for obtaining 

a contiguous frame group by detecting a plurality of frames that 
25 represent contiguous scenes in a video image; 

correspondent relationship acquisition means for 
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placing a reference patch comprising one or a plurality of 
rectangular areas on one of said plurality of frames included 
in said contiguous frame group which is used as a reference frame, 
then respectively placing on the others of said plurality of 
5 frames patches which are the same as said reference patch, then 

moving and/or deforming said patches in said other frames so 
that an image within the patch of each of said other frames 
approximately coincides with an image within said reference patch, 
and respectively acquiring correspondent relationships between 
10 pixels within the patches of said other frames and a pixel within 

said reference patch of said reference frame, based on the patches 
of said other frames after the movement and/or deformation and 
on said reference patch; and 

frame synthesis means for acquiring a synthesized frame 
15 from said plurality of frames, based on said correspondent 

relationships acquired by said correspondent relationship 
acquisition means . 

63. The video image synthesizer as set forth in claim 
62, wherein 

20 said contiguous frame group detection means is equipped 

with correlation computation means for computing a correlation 
between adjacent frames which is started from said reference 
frame; and 

said contiguous frame group, which is detected by said 
25 contiguous frame group detection means , comprises frames ranging 

from said reference frame to a frame, which is closer to said 
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reference frame, between a pair of said adjacent frames in which 
said correlation is lower than a predetermined first threshold 
value . 

64. The video image synthesizer as set forth in claim 

63, wherein 

said correlation computation means computes a 
histogram for at least one of the Y, Cb, and Cr components of 
each of said adjacent frames (where the Y component is a luminance 
component and the Cb and Cr components are color difference 
components) , also computes a Euclidean distance for each 
component between said adjacent frames by employing said 
histogram, and computes the sum of the Euclidean distances for 
said three components; and 

when said sum is a predetermined second threshold value 
or greater, said contiguous frame group detection means judges 
that said correlation between said adjacent frames is lower than 
said predetermined first threshold value. 

65. The video image synthesizer as set forth in claim 

64, wherein said correlation computation means computes said 
histogram by dividing each of components, which are used, among 
said three components by a value greater than 1 . 

66. The video image synthesizer as set forth in claim 
63, wherein 

said correlation computation means computes a 
difference between pixel values of corresponding pixels of said 
adjacent frames and also computes the sum of absolute values 



of the differences for all corresponding pixels; and 

when said sum is a third threshold value or greater, 
said contiguous frame group detection means judges that said 
correlation between adjacent frames is lower than said 
predetermined first threshold value. 

67. The video image synthesizer as set forth in claim 
63 r wherein said correlation computation means computes said 
correlation, employing a reduced image or thinned image of each 
frame . 

68. The video image synthesizer as set forth in claim 
62, further comprising stoppage means for stopping the detection 
of frames, which constitute said contiguous frame group, when 
the .number of frames detected by said contiguous frame group 
detection means reaches a predetermined upper limit value. 

69. A program for causing a computer to execute: 

a contiguous frame group detection process of obtaining 
a contiguous frame group by detecting a plurality of frames that 
represent contiguous scenes in a video image; 

a correspondent relationship acquisition process of 
placing a reference patch comprising one or a plurality of 
rectangular areas on one of said plurality of frames included 
in said contiguous frame group which is used as a reference frame, 
then respectively placing on the others of said plurality of 
frames patches which are the same as said reference patch, then 
moving and/or deforming said patches in said other frames so 
that an image within the patch of each of said other frames 
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approximately coincides with an image within said reference patch, 
and respectively acquiring correspondent relationships between 
pixels within the patches of said other frames and a pixel within 
said reference patch of said reference frame, based on the patches 
of said other frames after the movement and/or deformation and 
on said reference patch; and 

a frame synthesis process of acquiring a synthesized 
frame from said plurality of frames, based on said correspondent 
relationships . 

70. A video image synthesis method comprising the 

steps of: 

extracting a frame group that constitutes one or more 
important scenes from a video image; 

determining a frame, which is located at approximately 
a center, among a plurality of frames of said frame group as 
a reference frame for said important scene; 

placing a reference patch comprising one or a plurality 
of rectangular areas on said reference frame; 

respectively placing patches which are the same as said 
reference patch, on the others of said plurality of frames; 

moving and/or deforming said patches in said other 
frames so that an image within the patch of each of said other 
frames approximately coincides with an image within said 
reference patch; 

respectively acquiring correspondent relationships 
between pixels within the patches of said other frames and a 
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pixel within said reference patch of said reference frame, based 
on the patches of said other frames after the movement and/or 
deformation and on said reference patch; and 

acquiring a synthesized frame from said plurality of 
5 frames, based on said correspondent relationships. 

71. A video image synthesis method comprising the 

steps of: 

extracting a frame group that constitutes one or more 
important scenes from a video image; 
10 extracting high-frequency components of each of a 

plurality of frames constituting said frame group; 

computing the sum of said high-frequency components 
for each of said frames; 

determining a frame, in which said sum is highest, as 
15 a reference frame for said important scene; 

placing a reference patch comprising one or a plurality 
of rectangular areas on said reference frame; 

respectively placing patches which are the same as said 
reference patch, on the others of said plurality of frames; 
20 moving and/or deforming said patches in said other 

frames so that an image within the patch of each of said other 
frames approximately coincides with, an image within said 
reference patch; 

respectively acquiring correspondent relationships 
25 between pixels within the patches of said other frames and a 

pixel within said reference patch of said reference frame, based 



182 



on the patches of said other frames after the movement and/or 
deformation and on said reference patch; and 

acquiring a synthesized frame from said plurality of 
frames, based on said correspondent relationships. 

72. The synthesis method as set forth in claim 70, 

wherein 

a correlation between adjacent frames of said video 
image is computed; and 

a set of contiguous frames where said correlation is 
high is extracted as said frame group that constitutes one or 
more important scenes. 

73. The synthesis method as set forth in claim 71, 

wherein 

a correlation between adjacent frames of said video 
image is computed; and 

a set of contiguous frames where said correlation is 
high is extracted as said frame group that constitutes one or 
more important scenes. 

74. The synthesis method as set forth in claim 70, 

wherein 

a correlation between adjacent frames of said video 
image is computed; 

a set of contiguous frames where said correlation is 
high is extracted as a frame group that constitutes temporary 
important scenes ; 

correlations between said temporary important scenes 
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not adjacent are respectively computed; and 

a frame group, interposed between two temporary 
important scenes where said correlation is high and which are 
closest to each other, is extracted as said frame group that 
constitutes one or more important scenes. 

75. The synthesis method as set forth in claim 71, 

wherein 

a correlation between adjacent frames of said video 
image is computed; 

a set of contiguous frames where said correlation is 
high is extracted as a frame group that constitutes temporary 
important scenes; 

correlations between said temporary important scenes 
not adjacent are respectively computed; and 

a frame group, interposed between two temporary 
important scenes where said correlation is high and which are 
closest to each other, is extracted as said frame group that 
constitutes one or more important scenes. 

76. A video image synthesizer comprising: 
important scene extraction means for extracting a frame 

group that constitutes one or more important scenes from a video 
image; 

reference-frame determination means for determining 
a frame, which is located at approximately a center, among a 
plurality of frames of said frame group as a reference frame 
for said important scene; 
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correspondent relationship acquisition means for 
placing a reference patch comprising one or a plurality of 
rectangular areas on said reference frame, then respectively 
placing on the others of said plurality of frames patches which 
are the same as said reference patch, then moving and/ or deforming 
said patches in said other frames so that an image within the 
patch of each of said other frames approximately coincides with 
an image within said reference patch, and respectively acquiring 
correspondent relationships between pixels within the patches 
of said other frames and a pixel within said reference patch 
of said reference frame, based on the patches of said other frames 
after the movement and/or deformation and on said reference patch; 
and 

frame synthesis means for acquiring a synthesized frame 

from said plurality of frames, based on said correspondent 

relationships . 

77. A video image synthesizer comprising: 
important scene extractionmeans for extracting a frame 

group that constitutes one or more important scenes from a video 

image; 

reference-frame determination means for extracting 
high-frequency components of each of a plurality of frames 
constituting said frame group, then computing the sum of said 
high-frequency components for each of said frames, and 
determining a frame, in which said sum is highest, as a reference 
frame for said important scene; 
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correspondent relationship acquisition means for 
placing a reference patch comprising one or a plurality of 
rectangular areas on said reference frame, then respectively 
placing on the others of said plurality of frames patches which 
are the same as said reference patch, then moving and/or deforming 
said patches in said other frames so that an image within the 
patch of each of said other frames approximately coincides with 
an image within said reference patch, and respectively acquiring 
correspondent relationships between pixels within the patches 
of said other frames and a pixel within said reference patch 
of said reference frame, based on the patches of said other frames 
after the movement and/or deformation and on said reference patch; 
and 

frame synthesis means for acquiring a synthesized frame 
from said plurality of frames, based on said correspondent 
relationships . 

78. The video image synthesizer as set forth in claim 

76, wherein said important scene extraction means is equipped 
with correlation computation means for computing a correlation 
between adjacent frames of said video image, and extracts a set 
of contiguous frames, in which said correlation computed by said 
correlation computation means is high, as said frame group that 
constitutes one or more important scenes. 

79. The video image synthesizer as set forth in claim 

77, wherein said important scene extraction means is equipped 
with correlation computation means for computing a correlation 
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between adjacent frames of said video image, and extracts a set 
of contiguous frames, in which said correlation computed by said 
correlation computation means is high, as said frame group that 
constitutes one or more important scenes . 

80. The video image synthesizer as set forth in claim 

76, wherein 

said important-scene extraction means comprises 
first correlation computation means for computing a 
correlation between adjacent frames of said video image, 
temporary important scene extraction means for 
extracting a set of contiguous frames, in which said correlation 
computed by said first correlation computation means is high, 
as a frame group that constitutes temporary important scenes, 
and 

second correlation computation means for respectively 
computing correlations between said temporary important scenes 
not adjacent; and 

said important-scene extraction means extracts a frame 
group, interposed between two temporary important scenes where 
said correlation computed by said second correlation commutation 
means is high and which are closest to each other, as said frame 
group that constitutes one or more important scenes. 

81. The video image synthesizer as set forth in claim 

77, wherein 

said important-scene extraction means comprises 
first correlation computation means for computing a 
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correlation between adjacent frames of said video image, 
temporary important scene extraction means for 
extracting a set of contiguous frames, in which said correlation 
computed by said first correlation computation means is high, 
5 as a frame group that constitutes temporary important scenes, 

and 

second correlation computation means for respectively 
computing correlations between said temporary important scenes 
not adjacent; and 

10 said important-scene extraction means extracts a frame 

group, interposed between two temporary important scenes where 
said correlation computed by said second correlation commutation 
means is high and which are closest to each other, as said frame 
group that constitutes one or more important scenes. 

15 82. The video image synthesizer as set forth in claim 

76, wherein said important-scene extraction means comprises the 
important-scene extraction means as set forth in claim 78 and 
the important-scene extraction means as set forth in claim 80, 
and which further comprises selection means for selecting either 

20 the important-scene extraction means as set forth in claim 78 

or the important-scene extraction means as set forth in claim 
80. 

83. The video image synthesizer as set forth in claim 

77, wherein said important-scene extraction means comprises the 
25 important-scene extraction means as set forth in claim 78 and 

the important-scene extraction means as set forth in claim 80, 



188 



and which further comprises selection means for selecting either 
the important-scene extraction means as set forth in claim 78 
or the important-scene extraction means as set forth in claim 
80. 

5 84. A program for causing a computer to execute: 

an important scene extraction process of extracting 
a frame group that constitutes one or more important scenes from 
a video image; 

a reference-frame determination process of 

10 determining a frame, which is located at approximately a center, 

among a plurality of frames of said frame group as a reference 
frame for said important scene; 

a correspondent relationship acquisition process of 
placing a reference patch comprising one or a plurality of 

15 rectangular areas on said reference frame, then respectively 

placing on the others of said plurality of frames patches which 
are the same as said reference patch, then moving and/or deforming 
said patches in said other frames so that an image within the 
patch of each of said other frames approximately coincides with 

20 an image within said reference patch, and respectively acquiring 

correspondent relationships between pixels within the patches 
of said other frames and a pixel within said reference patch 
of said reference frame, based on the patches of said other frames 
after the movement and/or deformation and on said reference patch; 

25 and 

a frame synthesis process of acquiring a synthesized 
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frame from said plurality of frames, based on said correspondent 
relationships . 

85. A program for causing a computer to execute: 

an important scene extraction process of extracting 
a frame group that constitutes one or more important scenes from 
a video image; 

a reference-frame determination process of extracting 
high-frequency components of each of a plurality of frames 
constituting said frame group, then computing the sum of said 
high-frequency components for each of said frames, and 
determining a frame, in which said sum is highest, as a reference 
frame for said important scene; 

a correspondent relationship acquisition process of 
placing a reference patch comprising one or a plurality of 
rectangular areas on said reference frame, then respectively 
placing on the others of said plurality of frames patches which 
are the same as said reference patch, then moving and/or deforming 
said patches in said other frames so that an image within the 
patch of each of said other frames approximately coincides with 
an image within said reference patch, and respectively acquiring 
correspondent relationships between pixels within the patches 
of said other frames and a pixel within said reference patch 
of said reference frame, based on the patches of said other frames 
after the movement and/or deformation and on said reference patch; 
and 

a frame synthesis process of acquiring a synthesized 
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frame from said plurality of frames, based on said correspondent 
relationships . 

86. A method of acquiring a processed frame by 
performing image processing on a desired frame sampled from a 
video image, said method comprising the steps of: 

computing a similarity between said desired frame and 
at least one frame which is temporally before and after said 
desired frame; and 

acquiring said processed frame by obtaining a weighting 
coefficient that becomes greater if said similarity becomes 
greater, then weighting said at least one frame with said weighting 
coefficient, and synthesizing said weighted frame and said 
desired frame. 

87. The synthesis method as set forth in claim 86, 

wherein 

said desired frame is partitioned into a plurality of 

areas; 

said similarity is computed for each of corresponding 
areas in said at least one frame which correspond to saidplurality 
of areas; and 

saidprocessed frame is acquiredby obtaining weighting 
coefficients that become greater if said similarity becomes 
greater, then weighting said corresponding areas of said at least 
one frame with said weighting coefficients, and synthesizing 
said weighted areas and said plurality of areas. 

88. The synthesis method as set forth in claim 86, 
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wherein 

said desired frame is partitioned into a plurality of 
subject areas that are included in said desired frame; 

said similarity is computed for each of corresponding 
5 subject areas in said at least one frame which correspond to 

said plurality of subject areas; and 

saidprocessed frame is acquiredby obtaining weighting 
coefficients that become greater if said similarity becomes 
greater, then weighting said corresponding subject areas of said 
10 at least one frame with said weighting coefficients, and 

synthesizing said weighted subject areas and said plurality of 
subject areas. 

89 . An image processor for acquiring aprocessed frame 
by performing image processing on a desired frame sampled from 
15 a video image, said image processor comprising: 

similarity computation means for computing a 
similarity between said desired frame and at least one frame 
which is temporally before and after said desired frame; and 

synthesis means for obtaining a weighting coefficient 
20 that becomes greater if said similarity becomes greater, then 

weighting said at least one frame with said weighting coefficient, 
and synthesizing said weighted frame and said desired frame into 
said processed frame. 

90. The image processor as set forth in claim 89, 

25 wherein 

said similarity computation means partitions said 
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desired frame into a plurality of areas and computes said 
similarity for each of corresponding areas in said at least one 
frame which correspond to said plurality of areas; and 

said synthesis means obtains weighting coefficients 
that become greater if said similarity becomes greater, then 
weights said corresponding areas of said at least one frame with 
said weighting coefficients, and synthesizes said weighted areas 
and said plurality of areas into said processed frame. 

91. The image processor as set forth in claim 89, 

wherein 

said similarity computation means partitions said 
desired frame into a plurality of subject areas that are included 
in said desired frame, and computes said similarity for each 
of corresponding subject areas in said at least one frame which 
correspond to said plurality of subject areas; and 

said synthesis means obtains weighting coefficients 
that become greater if said similarity becomes greater, then 
weights said corresponding subject areas of said at least one 
frame with said weighting coefficients, and synthesizes said 
weighted subject areas and said plurality of subject areas into 
said processed frame. 

92. A program for causing a computer to execute an 
image processing method of acquiring a processed frame by 
performing image processing on a desired frame sampled from a 
video image, said program comprising: 

a similarity computation procedure of computing a 
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similarity between said desired frame and at least one frame 
which is temporally before and after said desired frame; and 

a synthesis process of obtaining a weighting 
coefficient that becomes greater if said similarity becomes 
5 greater, then weighting said at least one frame with said weighting 

coefficient, and synthesizing said weighted frame and said 
desired frame into said processed frame. 

93. The program as set forth in claim 92, wherein 

said similarity computation procedure comprises a 
10 procedure of partitioning said desired frame into a plurality 

of areas and a procedure of computing said similarity for each 
of corresponding areas in said at least one frame which correspond 
to said plurality of areas; and 

said synthesis procedure is a procedure of obtaining 
15 weighting coefficients that become greater if said similarity 

becomes greater, then weighting said corresponding areas of said 
at least one frame with said weighting coefficients, and 
synthesizing said weighted areas and said plurality of areas 
into said processed frame. 
20 94. The program as set forth in claim 92, wherein 

said similarity computation procedure comprises a 
procedure of partitioning said desired frame into a plurality 
of subject areas that are included in said desired frame, and 
a procedure of computing said similarity for each of corresponding 
25 subject areas in said at least one frame which correspond to 

said plurality of subject areas; and 
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said synthesis procedure is a procedure of obtaining 
weighting coefficients that become greater if said similarity 
becomes greater, then weighting said corresponding subject areas 
of said at least one frame with said weighting coef f icients, 
5 and synthesizing said weighted subject areas and said plurality 

of subject areas into said processed frame. 
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